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BACKGROUND OF THE INVENTION 

1. FIELD OF THE INVENTION: 

The present invention relates to an image 
processing apparatus for processing image data obtained 
by a CCD (charge-coupled device) area sensor having a 
primary-color filter so as to obtain a high quality image. 
Such an image processing apparatus is mounted on a digital 
camera (e.g., electronic still camera) or the like. 

2. DESCRIPTION OF THE RELATED ART: 

A conventional image processing apparatus for use 
in a digital camera, or the like, performs a series of 
various image processing steps^ e.g., interpolation for 
each color component, contour emphasizing processing, 
etc., on image data obtained by a CCD area sensor having 
a primary-color filter, so as to obtain a high quality 
image. Hereinafter, conventional image processing 
apparatuses are described with reference to Figures 13 
through 21. 

Figure 13 is a block diagram showing a first 
exemplary structure of a conventional image processing 
apparatus. In Figure 13, the image processing 
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apparatus 100 includes an optical low-pass filter 101, 
a primary-color CCD area sensor 102, a RGB interpolation 
section 103, a white balance adjustment section 104, a 
gamma correction section 105, and a sharpening processing 
section (contour emphasizing section) 106 • 

The low-pass filter 101 removes frequency 
components whose frequency is equal to or higher than a 
1/2 of sampling frequency fs. According to the sampling 
theorem, when an image is converted into image data, 
frequency components whose frequency is equal to or higher 
than a 1/2 of sampling frequency fs are converted into 
aliasing noise. In order to reduce such aliasing noise, 
the optical low-pass filter 101 is provided before the 
CCD area sensor 102. 

The primary-color CCD area sensor 102 is formed 
by a plurality of light -receiving elements arranged in 
a matrix. The primary-color CCD area sensor 102 includes 
a primary-color (R,G,B) filter on a light-receiving face 
thereof where each of color areas corresponds to one pixel. 
The color filter is formed based on the Bayer array 
{Figure 14) which is a type of RGB arrangements. 
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The RGB interpolation section 103 (which will be 
described later in detail) can receive, from the CCD area 
sensor 102, only one type of color component for each 
pixel among R, G, B colors of the color filter of the 
sensor 102. In the RGB interpolation section 103, each 
of the remaining two types of color components in each 
pixel is calculated from color components of the same type 
in neighboring pixels, whereby all of the three color 
components are obtained in each pixel. 

The white balance adjustment section 104 adjusts 
the white balance with respect to the R-, G-, and B- 
components in each pixel which are obtained by the RGB 
interpolation section 103 according to the color 
temperature of light so as to correct the color of an image. 

The gamma correction section 105 processes the R- , 
G-, and B-components obtained after the white balance has 
been adjusted such that the R-, G-, and B-components 
conform to properties of a display or printer which 
output s an image . 

The sharpening processing section 106 performs 
contour emphasizing processing (sharpening processing) 
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in order to obtain a sharp image. The sharpening process 
compensates for high-range luminance components which 
have been removed by the low-pass filter 101 by filtering 
processing which emphasizes high-range luminance 
components. That is, in the case where the low-pass 
filter 101 is provided, CCD image data generated by the 
primary-color CCD area sensor 102 has a reduced high- 
range luminance component. Furthermore, for example, 
since an unobtained color component in each pixel is 
compensated for by interpolation, e.g., by calculating 
an average value from color components of the same type 
in neighboring pixels, an image obtained by such 
interpolation has a further reduced high-range luminance 
component. Thus, in order to obtain a sharp image, it 
is necessary to compensate for a lost high-range luminance 
component by performing a filtering process which 
emphasizes high-range luminance components. 

Interpolation by the RGB interpolation 
section 103 is now described in detail. 

When the CCD area sensor 102 is a single-plate 
area sensor, in the RGB interpolation section 103, 
interpolation is performed for the missing two types of 
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color components such that all of R(red)-, G(green)-, and 
B(blue) -color components are complete In each pixel. 
Based on all of and B- color components including 

color components obtained by interpolation, a color image 
is created. 

When the CCD area sensor 102 is a three-plate area 
sensor where three area sensor elements are provided with 
full-size and B-color filters, respectively, the 

RGB interpolation section 103 receives from the CCD area 
sensor 102 all of R-, G- , and B-color components for each 
pixel . Thus , it is not necessary to perform interpolation . 
However, when the CCD area sensor 102 is a single-plate 
area sensor having, on its light -receiving element, a 
single color filter on which R-, G- , and B-f liter areas 
are arranged in a predetermined pattern (e.g., Bayer array 
shown in Figure 14), it is necessary to perform 
interpolation for missing color components in each pixel. 
Therefore, when employing a single-plate CCD camera 
incorporating a single-plate CCD area sensor, missing 
color components in each pixel which cannot be obtained 
by a color filter are created by various methods. 

U.S. Patent Nos . 4,605,956, 4,642,678, and 
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4,630,307, and the document by James E. Adams, Jr., 
"Interactions between color plane interpolation and other 
image processing functions in electronic photography", 
suggest various interpolation methods for the purpose of 
creating an image with no jaggy or zip noise. "Jaggy" 
means a step- shaped noise which emerges along a contour 
of an image, which is a type of noise among various types 
of noise. 

In U.S. Patent No. 4,605,956, interpolation is 
performed by using the following expressions (1) 
through (4) on missing color components in a pixel 
arrangement shown in Figure 15 (bilinear interpolation 
method) . Interpolation for G- and B-components at each 
pixel position in Figures 14 and 15 is shown below: 

G5 = (G2+G4+G6+G8)/4 .-.(1) 

B5 = (Bl+B3+B7+B9)/4 -..(2) 

B2 = (Bl+B3)/2 ...(3) 

B4 = (Bl+B7)/2 ... (4) 

In the Bayer array shown in Figure 15, R- , G-, and 
B-colors are arranged according to a certain pattern • For 
example, at a central pixel position, R5, color 
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components G5 and B5 are missing in a corresponding color 
filter • The color components G5 and B5 are obtained from 
expressions (1) and (2). 

Furthermore, B2 is obtained from expression (3) , 
and B4 is obtained from expression (4). Interpolation 
for each of B6 and B8 is performed by obtaining an average 
value of B- component values in vertically or horizontally 
neighboring pixels based on an expression similar to 
expression (3) or (4). This is the same for R2, R4, R6, 
and R8. 

Furthermore, Gl is obtained from G-component 
information in neighboring pixels around pixel position 
"Bl" by an expression similar to expression (1). This 
is the same for G3, G7, and G9. Rl is obtained from 
R-component information in neighboring pixels around 
pixel position "Bl" by an expression similar to 
expression (2). This is the same for R3, R7 , and R9 . 

Next, an image processing method disclosed in 
Japanese Laid-Open Publication No. 10-164371 is 
described with reference to Figure 16. 
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Figure 16 is a block diagram showing a second 
exemplary structure of a conventional image processing 
apparatus. As shown in Figure 16^ the image processing 
apparatus 200 includes a primary-color CCD area 
sensor 201, a RGB interpolation section 202, and a 
contour emphasizing sections 203 provided for respective 
ones of G-, and B-components . 

The RGB interpolation section 202 first 
interpolates a G-component by using expression (5) which 
represents median processing for G-color components 
obtained by the filter arrangement shown in Figure 15. 
Then, only at pixel positions, "R" and "B", in Figure 14, 
a (R-G) component and a (B-G) component are created. In 
the last, the created (R-G) and (B-G) components are 
interpolated by bilinear interpolation, and G-component s 
are added to the interpolated (R-G) and (B-G) components, 
whereby R- and B-components are obtained. 

G5 = (G2+G4+G6+G8-Min-Max)/2 ..-(5) 
Min = Min(G2,G4,G6,G8) 
Max = Max{G2,G4,G6,G8) 

How to obtain B4 at pixel position "G4" is now 
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described in detail with reference to Figure 15. In the 
first step, Gl, G3, G5, G7. and G9 are interpolated by 
the median processing process using expression (5). 
Then, (Bl-Gl) and (B7-G7) which are (B-G) components at 
pixel positions "Bl" and "B7" are created. (B4-G4) 
which is a (B-G) component at pixel positions "G4" is 
represented as follows: 

B4-G4 = (1/2){(B1-G1)+(B7-G7)} 
From this expression, B4 can be represented as follows: 
B4 = (1/2){(B1-G1)+(B7-G7)}+G4 

The contour emphasizing sections 203 employs a 
two-dimensional second-derivative filter shown in 
Figure 17A for each of R-, G-, and B-components . Each 
box of the two-dimensional second- derivative filter shown 
in Figure 17A corresponds to one pixel, and the number 
shown in each box represents a weight . The weight of each 
box area is set such that the total weight in the 
two-dimensional second-derivative filter is zero. 

Next, an interpolation method disclosed in 
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Japanese Laid-Open Publication No. 11-18047 is described 
with reference to Figure 18. 

Figure 18 is a block diagram showing a third 
5 exemplary structure of a conventional image processing 
apparatus. In Figure 18, the image processing 
apparatus 300 includes a primary-color CCD area 
sensor 301, RGB interpolation sections 302, a middle- 
range component emphasizing section 303, a high-range 
10 component emphasizing section 304, a white balance 
adjustment section 305, and a gamma correction 
section 306. The primary-color CCD area sensor 301 
includes a color filter of Bayer array. 

15 The RGB interpolation sections 302 perform 

interpolation on each of R-, G-, and B-components . When 
a G-component is interpolated by using a median method 
of expression (5), and R- and B-components are 
interpolated by using a bilinear method of 

20 expressions (2) through (4), interpolation is achieved 
with high image quality. 

The middle-range component emphasizing 
section 303 is formed by a middle-range component 
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extraction section 303a and adders 303b. In the 
middle-range component emphasizing section 303, the 
middle-range component extraction section 303a extracts 
a middle-range component from a G-component interpolated 
5 by the G- interpolation section of the RGB interpolation 
sections 302. The extracted middle-range component is 
synthesized with each of R- , G- , and B-components by the 
adders 303b, whereby the middle -range component 
emphasizing processing is achieved. In this middle- 

10 range component emphasizing processing, the difference 
between a G-component and a low-pass filtered G-component 
is added as a compensation component to the G-component. 
In other words, a high-range component of the G-component 
is removed by a low-pass filter (not shown) to obtain a 

15 GBAR - component , and a compensation component which is the 
difference between the G-component and the GBAR -component 
(G-GBAR) is added to the G-component, whereby a 
middle-range component emphasizing processing is 
achieved. 

20 

The high-range component emphasizing section 304 
is formed by a high- range component extraction 
section 304a and adders 304b. In the high-range 
component emphasizing section 304, the high-range 
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component extraction section 304a extracts a high-range 
component from a G-component interpolated by the <3- 
interpolation section of the RGB interpolation 
sections 302, The extracted high-range component is 
5 synthesized with each of R- , G-, and B-components by the 
adders 304b, whereby high-range component emphasizing 
processing is achieved. This high-range component 
emphasizing process employs the two-dimensional 
second- derivative filter shown in Figure 17B. In the 

10 filter shown in Figure 17B, in order to obtain pixel data 
of a pixel corresponding to a central box "4", the weight 
of the filter is multiplied by data from each pixel, and 
the products for all of the pixels are added up. Thus, 
when a result of this calculation does not change, the 

15 sum of the products results in zero. When a result of 
this calculation changes much, the sum becomes a large 
value, whereby a high- range component is emphasized. 

Now, consider a case where each of R- , G-, and 
20 B-components obtained through a Bayer array filter shown 
in Figure 14 is sampled at a sampling 
frequency f s=l/Ax=l/Ay . A sampling frequency 

distribution range obtained in this case is shown in 
Figure 19, Herein, "Ax" denotes the width of a pixel 
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(pixel pitch) in a horizontal direction (x-direction) , 
and "Ay" denotes the width of a pixel (pixel pitch) in 
a vertical direction (y-direction) • 

5 According to the sampling theorem, a highest 

restorable frequency within a spatial frequency band of 
an original image is represented by a solid-line lozenge 
which is indicated by an arrow " G- component " in Figure 19 . 
Furthermore, R- and B- components are included in a two -dot 

10 chain line square • Thus, a frequency band which can be 
accurately converted into R-, G-, and B-components of 
image data is an area within the square defined by the 
two-dot chain line. As seen from Figure 19, the 
accurately-restorable, highest frequency of the original 

15 image is a half of the sampling frequency f s ( =1/Ax=l/Ay ) . 
Therefore, a frequency component which is higher than the 
highest restorable frequency fs/2 emerges as a noise in 
the image data. 

20 In general, for the purpose of avoiding such a 

problem, an optical low-pass filter (anti-aliasing 
filter) is provided to a CCD area sensor • This low-pass 
filter removes frequency components of image data which 
are higher than the highest restorable frequency fs/2. 
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but undesirably attenuates some of frequency components 
which are lower than the highest restorable 
frequency fs/2 because the low-pass filter is not ideal. 
In Figure 20, graph a shows an ideal low-pass filter 
5 frequency characteristic (i.e., only frequency 
components which are higher than the highest restorable 
frequency fs/2 are removed) . However, in an actual case, 
the low-pass filter frequency characteristic results in 
a curve of graph b. Furthermore, graph o shows an ideal 

10 frequency characteristic of a compensation filter which 
is used to compensate for the characteristic of graph b 
so as to obtain a characteristic similar to the ideal 
characteristic of graph a. According to the present 
invention, as described later in detail, a middle-range 

15 luminance component and a high-range luminance component 
are synthetically added to a newly- extracted, middle- 
high range luminance component according to a 
predetermined synthetic ratio, whereby a low-pass 
filtered component is compensated for by using a 

20 compensation filter having a frequency characteristic 
similar to graph o. 

Furthermore, when interpolation is performed on 
components of an image which have been obtained through 
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a Bayer array filter shown in Figure 14 so as to compensate 
for the missing two types of color components in each pixel, 
high-range luminance components are attenuated. 
Accordingly, it is indispensable for generating a sharp 
5 image to compensate for attenuated high-range components . 
In general, such a compensation is achieved by separately 
performing a compensation for middle-range luminance 
components with a compensation filter having a frequency 
characteristic of graph d and a compensation for 

10 high-range luminance components with a compensation 
filter having a frequency characteristic of graph c. In 
Figure 21, graph b shows a synthetic frequency 
characteristic obtained after low-pass filtering (with 
an anti-aliasing filter) and interpolation processing, 

15 and graph a shows an ideal frequency characteristic 
obtained after an entire process in an image processing 
system including compensation processing. 



In the above-described conventional art, 
20 attenuated middle- and high- range components among 
low-pass filtered components are compensated for by 
compensation filters. The middle-range components are 
compensated for by a compensation filter having a 
characteristic represented by graph d of Figure 21, and 
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the high-range components are compensated for by a 
compensation filter having a characteristic represented 
by graph o of Figure 21. When graph d exhibits the 
maximum amplitude, the angular frequency co is jt;/2 
5 (equivalent to fs/4) . When graph c exhibits the maximum 
amplitude, the angular frequency o) is Jt (equivalent to 
fs/2) • 



However, in contour emphasizing processing which 
10 is performed for obtaining a sharp image, when high-range 
components of an image are emphasized by compensating for 
high-range components, all components having an angular 
frequency w higher than fs/2 are converted into noise. 
Thus, as the resolution of the image is increased, noise 
15 and jaggy are observed more prominently. 



For example, in the method described in connection 
with the first exemplary conventional structure shown in 
Figure 13, a G- component which largely contributes to the 
20 luminance of an image is interpolated according to above 
expression (1), and image data obtained after the 
interpolation is subjected to contour emphasizing 
processing, and as a result, jaggy emerges in an edge 
portion of the image. Examples of proposed method for 
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solving such a problem include a color- smoothing 
interpolation method {U»S. Patent No* 4,642,678), a 
pattern-recognition interpolation method ( U • S • Patent 
No, 4,630,307), an adaptive interpolation method 
5 (document by James E. Adams, Jr.). These methods are 
different interpolation methods, but all performed in a 
structure similar to the image processing apparatus 100 
shown in Figure 13 . That is , a high-range component which 
is indispensable for obtaining a sharp image is generated 

10 by contour emphasizing processing in the sharpening 
processing section 106 which resides at the end of the 
entire process . These interpolation methods each have 
some advantages in reducing false color and jaggy. 
However, these interpolation methods do not have means 

15 for processing a high-range component except for 
high-range emphasizing filtering, and thus, noise occurs 
in the high-range emphasizing filtering. 



Thus, in the methods described in connection with 
20 the above first through third exemplary conventional 
structures (Figures 13, 16, and 18), noise is inevitably 
emphasized when a high-range component is compensated for 
by sharpening processing. That is, during interpolation 
of high-range components, frequency components whose 
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frequency is equal to or higher than 1/2 of a sampling 
frequency fs are also emphasized, and the emphasized 
components whose frequency is equal to or higher than a 
1/2 of sampling frequency f s are observed as noise and/or 

5 jaggy- 

SUMMARY OF THE INVENTION 

According to one aspect of the present invention, 
10 an image processing apparatus for processing RGB image 
data output from an image capturing element including a 
primary-color filter includes: a middle-high range 
luminance component compensation section for 
compensating for a middle-high range luminance component 
15 of a low-frequency luminance signal generated based on 
the RGB image data such that the low- frequency luminance 
signal has substantially an ideal frequency luminance 
characteristic which is lower than or equal to a 
predetermined frequency. 

20 

In this specification, a "middle-high range 
luminance component" means a luminance component mainly 
containing middle-high range components, and a "low- 
frequency luminance signal" means a luminance signal 
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mainly containing low-frequency components • 

With the above structure, a middle -high range 
luminance component of a low- frequency luminance signal 
5 which is attenuated as compared with an ideal frequency 
characteristic in a range of a predetermined frequency 
(a 1/2 of sampling frequency f s) or smaller is compensated. 
Thus, a sharp image can be obtained by contour emphasizing 
processing while preventing occurrence of noise and jaggy 
% 10 which may be caused when obtaining the sharp image. 

b? f 

According to another aspect of the present 
'■^ invention, an image processing apparatus for processing 

y RGB image data output from an image capturing element 

n 15 including a primary-color filter includes: a middle-high 

O range luminance component extraction section for 

extracting a middle-high range luminance component which 
has a zero amplitude at an angular frequency o>=Jt and a 
maximum amplitude at an angular frequency o between jt/2 
20 and n from a first luminance signal generated based on 
RGB image data; and a first synthesis section for adding 
the middle -high range luminance component to a low- 
frequency luminance signal generated based on the RGB 
image data so as to generate a second luminance signal. 
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With such a structure, an image with high 
resolution can be obtained by contour emphasizing 
processing while preventing occurrence of noise and jaggy 
5 which may be caused when obtaining the sharp image. 

In one embodiment of the present invention, the 
middle-high range luminance component extraction section 
uses at least one filter having a size of an even-number 
10 of pixels to arithmetically process the first luminance 
signal . 

With such a structure, a middle-high range 
component whose amplitude is zero when the angular 
15 frequency co is Jt and has a maximum value at a position 
where the angular frequency o is between jt/2 to jt can be 
readily extracted. 

In another embodiment of the present invention, 
20 the filter having a size of an even-number of pixels is 
a two-dimensional filter and has coefficients 
symmetrically arranged with respect to a x-direction and 
a y-direction. 
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With such a structure, a uniform filtering effect 
can be obtained, and as a result , an image can be faithfully 
reproduced. 

5 In still another embodiment of the present 

invention, the filter having a size of an even- number of 
pixels includes a first low-pass filter having a 
differentiation capability and a second low-pass filter; 
and a difference between an output obtained by 
10 arithmetically processing the first luminance signal 
using the first low-pass filter and an output obtained 
by arithmetically processing the first luminance signal 
using the second low-pass filter is output as the 
middle-high range luminance component. 

15 

With such a structure, arithmetic operations in 
X- and y-directions of image data can be separately 
performed, and accordingly, increases in the amount of 
arithmetic operations can be suppressed. Thus, YH 
20 extraction filtering can be readily implemented by 
hardware . 

In still another embodiment of the present 
invention, the image processing apparatus further 
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includes: a first interpolation section for interpolating 
missing components among R- , G- , and B-components for each 
pixel before the generation of the first luminance signal, 
wherein the first interpolation section interpolates 
5 missing components by arithmetically processing the RGB 
image data using a filter having a size of 3 pixels x 
3 pixels. 

With such a structure, a middle-high range 
10 luminance component can be extracted while most- 
ef fectively preventing deterioration of the middle-high 
range luminance component . 

In still another embodiment of the present 
15 invention, the image processing apparatus further 
includes: a second interpolation section for 
interpolating missing components among R- , G- , and B- 
components for each pixel before the generation of the 
low- frequency luminance signal, wherein the second 
20 interpolation section interpolates missing components by 
arithmetically processing the RGB image data using a 
filter having a size of an even-number of pixels. 

With such a structure, when compensating for a 
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middle-high range luminance component of a low- frequency 
luminance signal, a center of the middle-high range 
luminance component is present at a position between 
neighboring pixels, and a center of the low-frequency 
5 luminance signal is also present at a position between 
neighboring pixels. Thus, occurrence of a ghost in a 
reproduced image can be prevented. 

In still another embodiment of the present 
10 invention, at least one of the first and second 
interpolation sections interpolates the RGB image data 
by using a median method for a G-component and a bilinear 
method for R- and B-components , 

15 With such a structure, a G-component is 

interpolated by using a median method, whereby 
attenuation of a high-range luminance component is 
suppressed to a minimum level. R- and B- components are 
interpolated by using a bilinear method, whereby noise 

20 is reduced. Thus, a contour of an image is emphasized, 
and the quality of the image can be improved. 

In still another embodiment of the present 
invention, the image processing apparatus further 
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includes: a middle/high- range luminance component 
extraction section for extracting at least one of a 
middle-range luminance component and a high-range 
luminance component based on the second luminance signal; 
5 and a second synthesis section for adding at least one 
of the middle-range luminance component and the high- 
range luminance component to the second luminance signal 
so as to generate a third luminance signal. 

10 In this specification, a "middle-range luminance 

component" means a luminance component mainly containing 
middle-frequency components , and a "high-range luminance 
component" means a luminance component mainly containing 
high-frequency components • 

15 

With the above structure by changing a ratio 
between a middle-range luminance component and a 
high-range luminance component, the three-dimensional 
appearance (stereoscopic effect or stereophonic effect) 
20 of an image can be adjusted according to user's 
preference . 

In still another embodiment of the present 
invention, wherein the middle/high-range luminance 
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component extraction section arithmetically processes 
the second luminance signal by using one filter which has 
an adjustable coefficient. 

5 With such a structure, a middle/high-range 

component extraction section can be readily formed of a 
single filter. 

In still another embodiment of the present 
10 invention, the image processing apparatus further 
includes: a median filtering section for removing, with 
a median filter, noise inherent to the image capturing 
element which is contained in a color-difference signal 
generated based on a RGB image signal from the second 
15 interpolation section; wherein the median filtering 
section changes the size of the median filter according 
to an amount of the noise. 

With such a structure, the amount of noise 
20 included in a color-difference signal varies according 
to, for example, the quality of an image capturing element 
such as a CCD. Thus, by selecting an appropriate median 
filter according to the amount of noise, a color- 
difference signal with reduced noise can be generated. 
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Thus , the invention described herein makes 
possible the advantages of providing an image processing 
apparatus which can prevent noise and jaggy which may 
5 occur when obtaining a sharp image. 

These and other advantages of the present 
invention will become apparent to those skilled in the 
art upon reading and understanding the following detailed 
10 description with reference to the accompanying figures. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Figure 1 is a block diagram showing a structure 
15 of an image processing apparatus according to an 
embodiment of the present invention. 

Figures 2A and 2B show specific examples of YH 
extraction filters used in a middle-high range luminance 
20 component extraction section of the image processing 
apparatus shown in Figure 1 . 

Figure 3 shows frequency characteristics of the 
YH extraction filters shown in Figures 2A and 2B and a 
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frequency characteristic of a differential output of the 
YH extraction filters shown in Figures 2A and 2B, 

Figure 4 shows a specific example of a composite 
5 contour emphasizing filter used in a middle/high range 
luminance component extraction section of the image 
processing apparatus shown in Figure 1. 

Figures 5A and 5B show RGB interpolation filters 
10 used in a second RGB interpolation section of the image 
processing apparatus shown in Figure 1. Figure 5A shows 
a specific example of an R- and B-interpolation filter • 
Figure 5B shows a specific example of a G-interpolation 
filter. 

15 

Figure 6 shows examples of one -dimensional 
filters in parts (A) through (G), 

Figure 7 shows frequency characteristics of 
20 filter A having a size of an even-number of pixels and 
filters B and C each having a size of an odd-number of 
pixels . 

Figure 8 shows frequency characteristics of 
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filter A having a size of an even-number of pixels and 
filters D and E each having a size of an odd-number of 
pixels . 

5 Figure 9 shows frequency characteristics of 

various filters each having a different size of an 
even-number of pixels. 

Figure 10 illustrates a case where pixel data is 
10 generated between neighboring pixels with a filter having 
a size of an even-number of pixels. 

Figure 11 shows frequency characteristics of a 
low-frequency luminance signal, a middle-high luminance 
15 component , and a low- frequency luminance signal including 
a compensated middle-high luminance component. 

Figure 12 shows frequency characteristics of a 
middle-high range luminance component extraction filter, 
20 a middle-range luminance component extraction filter, and 
a high-range luminance component extraction filter. 

Figure 13 is a block diagram showing a first 
exemplary structure of a conventional image processing 
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apparatus . 

Figure 14 is a plan view showing a Bayer- array of 
a color filter. 

Figure 15 shows a portion of the Bayer- array of 
Figure 14. 

Figure 16 is a block diagram showing a second 
exemplary structure of a conventional image processing 
apparatus . 

Figures 17A and 173 show two-dimensional 
second-derivative filters which are used in contour 
emphasizing processing. 

Figure 18 is a block diagram showing a third 
exemplary structure of a conventional image processing 
apparatus . 

Figure 19 shows a restorable range of sampling 
frequencies for each color component in the Bayer array. 

Figure 20 shows frequency characteristics of an 
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optical low-pass filter and a compensation filter. 

Figure 21 shows frequency characteristics of a 
middle-range luminance component, a middle-high range 
luminance component, and a high-range luminance component 
compensation filter. 

Figure 22 shows a specific example of a YH 
extraction filter formed of a single filter which is used 
in a middle-high luminance component extraction section 
of the image processing apparatus shown in Figure 1. 

DESCRIPTION OF THE PREFERRED EMBODIMENTS 

Hereinafter, an embodiment of the present 
invention will be described with reference to the 
drawings . 

Figure 1 is a block diagram showing a structure 
of an image processing apparatus according to an 
embodiment of the present invention. In Figure 1, the 
image processing apparatus 1 includes: an optical 
low-pass filter 2; a primary-color CCD area sensor 3; a 
first RGB interpolation section (RGB interpolation 
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section for extraction of middle-high range luminance 
component) 4; a luminance generation section 5 for 
extraction of middle-high range luminance component; a 
middle -high range luminance component extraction 
section 6; a multiplier 7 and adder 8 which function as 
a first synthesizing section; a middle/high range 
luminance component extraction section 9; and a 
multiplier 10 and adder 11 which function as a second 
synthesizing section. The first RGB interpolation 
section 4, the luminance generation section 5, and the 
middle -high range luminance component extraction 
section 6 form a middle-high range luminance component 
compensation section 20, 

The low-pass filter 2 removes frequency 
components whose frequency is equal to or higher than a 
1/2 of sampling frequency fs. 

The primary-color CCD area sensor 3 is formed by 
a plurality of light -receiving elements arranged in a 
matrix. The primary-color CCD area sensor 3 includes a 
color filter on a light-receiving face thereof. This 
color filter is formed based on the Bayer array 
(Figure 14). It should be noted that RGB image data 
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obtained from the primary-color CCD area sensor 3 
successively goes through a Correlated Double Sampling 
(CDS) circuit (not shown) which reduces noise in the RGB 
image data, an Automatic Gain Control (AGC) circuit (not 
5 shown) for performing gain adjustment processing, and an 
A/D conversion circuit (not shown) whose resolution is, 
e.g., 10 bits, and then reaches, as digital image data, 
the first RGB interpolation section 4 and a second RGB 
interpolation section (RGB interpolation section for 
10 generation of low-frequency luminance signal generation 
and color-difference signal) 12* (The second RGB 
interpolation section 12 will be described later in 
detail. ) 



15 In the first RGB interpolation section 4, missing 

color components among R- , G-, and B-components for each 
pixel are interpolated based on data from color components 
of the same type in neighboring pixels. In this example, 
the R- and B-components are interpolated by a bilinear 

20 method represented by expressions (2) through (4), and 
the G-component is interpolated by a median method 
represented by expressions (5). The first RGB 

interpolation section 4 uses, as an RGB interpolation 
filter, a two-dimensional filter having a size of (an 
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odd-number of pixels) x (an odd-number of pixels) which 
is greater than 3 pixels x 3 pixels . 

As the size of this interpolation filter becomes 
5 larger, a larger amount of middle- and high- luminance 
components are reduced by this filter. As a result, it 
becomes more difficult to extract the middle- and 
high- luminance components. Interpolation processing 
would not be achieved by a filter of one-pixel by one-pixel . 
10 Furthermore, if a two-dimensional filter of 2 pixels x 

2 pixels is employed, a bilinear interpolation method 
cannot be used, and therefore, interpolation cannot be 
achieved without using a nearest neighbor interpolation. 
When the nearest neighbor interpolation is used, middle- 

15 and high- luminance components are deteriorated as 
compared with a case where a bilinear interpolation method 
is used. Thus, a two-dimensional filter of 3 pixels x 

3 pixels is desirable. 

20 The luminance generation section 5 uses a first 

RGB image signal which is obtained by the first RGB 
interpolation section 4 so as to generate a luminance 
signal Y for extracting middle- and high- luminance 
components : 
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Y = 0.30R+0*59G+0.11B .(6) 

In the middle-high range luminance component 
5 extraction section 6, the luminance signal Y which is 
generated by the luminance generation section 5 is 
subjected to a first YH extraction filter Fl shown in 
Figure 2A and a second YH extraction filter F2 shown in 
Figure 2B, whereby middle-high range luminance 

10 component YH is extracted • Specifically, the middle- 
high range luminance component extraction section 6 
outputs a difference between an output from the first YH 
extraction filter Fl having a size of 6 pixels x 6 pixels 
shown in Figure 2A and an output from the second YH 

15 extraction filter F2 having a size of 4 pixels x 4 pixels 
shown in Figure 2B as middle-high range luminance 
component YH. In the YH extraction filters 1 and 2, x 
denotes an operation in a horizontal direction, and y 
denotes an operation in a vertical direction, 

20 

The first YH extraction filter Fl shown in 
Figure 2A is a low-path filter having a differential 
effect (which includes positive and negative 
coefficients). The first YH extraction filter Fl is 
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symmetric with respect to x- and y-directions , and may 
have a size of an even-number of pixels along both x- and 
y-directions (4 pixels x 4 pixels, 8 pixels x 8 pixels, 
10 pixels X 10 pixels, etc.). Furthermore, it is 
5 desirable that the first YH extraction filter Fl has a 
square shape (having a size of a same number of pixels 
along both x- and y-directions). However, the first YH 
extraction filter Fl may have a horizontally-oblong, 
rectangular shape where the size in the x-direction is 

10 larger than the size in the y-direction if the x-direction 
of an image is emphasized. Alternatively, the first YH 
extraction filter Fl may have a vertically-oblong, 
rectangular shape where the size in the y-direction is 
larger than the size in the x-direction if the y-direction 

15 of an image is emphasized. Furthermore, in the first YH 
extraction filter Fl, coefficients of terms are 
symmetrically arranged with respect to both x- and y- 
directions of the filter Fl. 

20 The second YH extraction filter F2 shown in 

Figure 2B is a low-path filter (which includes only 
positive coefficients). The second YH extraction 
filter F2 has a size of an even-number of pixels along 
both X- and y-directions (but smaller than the size of 
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the filter Fl) • Furthermore, it is desirable that the 
second YH extraction filter F2 has a square shape. 
However, the second YH extraction filter F2 may have a 
horizontally-oblong, rectangular shape or a 
5 vertically-oblong, rectangular shape. Furthermore, in 
the second YH extraction filter F2, coefficients of terms 
are symmetrically arranged with respect to both x- and 
y-directions of the filter F2. With such a symmetrical 
arrangement of coefficients, an effect of filtering can 
10 be uniformly obtained, and as a result, an image can be 
faithfully reproduced. 

With such a YH extraction filter formed by two 
filters, operations in x- and y-directions can be 

15 separately performed. Thus, YH extraction filtering can 
be readily implemented by hardware. Figure 3 shows 
frequency characteristics of the YH extraction filters Fl 
and F2 (Figures 2A and 2B) and a frequency characteristic 
of a differential output of the filters Fl and F2. In 

20 Figure 3, graph h shows a frequency characteristic of the 
first YH extraction filter Fl, graph i shows a frequency 
characteristic of the second YH extraction filter F2, and 
graph j shows a frequency characteristic of middle-high 
range luminance component YH which is the differential 
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output of graph ± and graph j. In this example, the YH 
extraction filter is formed by two filters, but may be 
formed by a single filter . In such a case , the arrangement 
of coefficients of the filter is as shown in Figure 22, 
5 and the amount of arithmetic operations is increased. 

The multiplier 7 gives supplemental compensation, 
by multiplying a gain a which is an adjustment coefficient , 
to middle-high range luminance component YH which has 
10 been extracted by the middle-high range luminance 
component extraction section 6, 

The adder 8 adds a middle -high range luminance 
component multiplied by a predetermined coefficient of 
15 gain a, aYH, to a low-frequency luminance signal YL which 
is obtained by a low- frequency luminance signal 
generation section 15 (described later), thereby 
generating a luminance signal (YL+aYH) . 

20 In the middle/high-range luminance component 

extraction section 9, a luminance signal (YL+aYH) 
received from the adder 8 is subjected to a composite 
contour emphasizing filter shown in Figure 4, whereby 
middle/high-range luminance components ENH are extracted. 
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In the filter of Figure 4, the ratio of extracted 
middle-range luminance components and extracted high- 
range luminance components can be adjusted by adjusting 
the ratio between variables 1 and m. 

5 

The multiplier 10 gives supplemental 
compensation, by multiplying a gain p which is an 
adjustment coefficient, to middle- and high-range 
luminance components ENH which has been extracted by the 
10 middle/high-range luminance component extraction 
section 9, 



The adder 11 adds middle- and high-range 
luminance components multiplied by the multiplier 10 by 
15 a predetermined coefficient of gain pENH, to a 
luminance signal (YL+aYH) which is obtained from the 
adder 8, and outputs a luminance signal ( YL+aYH+PENH) . 



The image processing apparatus 1 further 
20 includes : a second RGB interpolation section 12 connected 
to the primary-color CCD area sensor 3; a white balance 
adjustment section 13; a gamma correction section 14; a 
low-frequency luminance signal generation section 15; a 
color-difference signal generation section 16; and a 
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median filter 17* 

In the second RGB interpolation section 12, 
interpolation is achieved by arithmetic operations using 
5 interpolation filters shown in Figures 5A and 5B. Color 
components interpolated hy the second RGB interpolation 
section 12 and color components obtained by the color 
filter are combined so as to generate a second RGB image 
signal where all of R-, G-, and B-components are complete 

10 for each pixel. Specifically, the R- and B-component 
interpolation filter shown in Figure 5A is used for R- 
and B-components, and the G-component interpolation 
filter shown in Figure 5B is used for the G-component. 
Furthermore, in the G-component interpolation filter, it 

15 is required that an operation in the x-direction has 
priority over an operation in the y-direction. 

For example, when R-components are interpolated, 
only R-components in the Bayer array shown in Figure 14 
20 are input to the R- and B-component interpolation filter 
shown in Figure 5A, and zero is input for B- and G- 
components to the R- and B-component interpolation filter. 
As a result, an output from the R- and B-component 
interpolation filter results in an interpolated R- 
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component image. In the R- and B-component interpolation 
filter shown in Figure 5A, the order of arithmetic 
operations is not important because a weighting 
coefficient is the same for both the x- and y-directions . 
However, in the G-component interpolation filter shown 
in Figure 5B, an operation in the x-direction must be 
perf oarmed prior to an operation in the y-direction because 
a weighting coefficient for the x-direction is different 
from that for the y-direction. If an operation in the 
y-direction must be performed prior to an operation in 
the x-direction, weighting coefficients for the x- and 
y-directions shown in the G-interpolation filter of 
Figure 5B are replaced with each other. Each of the 
interpolation filters shown in Figures 5A and 5B is a 
low-path filter (which includes only positive 
coefficients). Each of these filters has a size of an 
even-number of pixels along both x- and y-directions. 
Furthermore, in each of these filters, coefficients of 
terms are symmetrically arranged with respect to both x- 
and y-directions of the filter. With such a symmetrical 
arrangement of coefficients, an effect of filtering can 
be uniformly obtained, and as a result, an image can be 
faithfully reproduced. 
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In the second RGB interpolation section 12, 
filters having a size of an even-number of pixels are 
employed because at the adder 8 where middle -high range 
luminance component YH is added to a low-frequency 
luminance signal YL, a position of image data from the 
section 15 must conform to that of image data from the 
multiplier 7, Specifically, as described above, since 
the YH extraction filter of the middle-high range 
luminance component extraction section 6 must have a size 
of an even-number of pixels whereas the interpolation 
filter used in the first RGB interpolation section 4 has 
a size of an odd-number of pixels, image data of the 
extracted middle -high range luminance component YH 
reside in positions between pixels. Therefore, new 
pixels must be created on boundaries between pixels in 
pixel data of the low-frequency luminance signal YL, and 
to this end, a filter having a size of an even-number of 
pixels is used in the second RGB interpolation section 12* 
In such an arrangement, pixel data of the low-frequency 
luminance signal YL is positioned between pixels so as 
to conform to a position of image data of the extracted 
middle-high range luminance component YH. 

The white balance adjustment section 13 adjusts 
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the white balance with respect to the interpolated R- , 
or B-components In each pixel according to the color 
temperature of light so as to correct the color of an Image • 

The gamma correction section 14 processes the 
and B-components obtained after the white balance has 
been adjusted such that the R-, and B-components 

conform to properties of a display or printer for 
outputtlng an Image. 

The low- frequency luminance signal generation 
section 15 generates a low- frequency luminance signal YL 
based on above expression (6) after gamma correction. 

The color-difference signal generation 
section 16 generates color-difference signals Cr and Cb 
based on expressions (7) after gamma correction: 

Cr = 0.70R-0.59G-0.11B 

Cb = -0.30R-0.59G+0.89B ...(7) 

The median filter 17 removes noise from the above 
color-difference signals Cr and Cb. The size of the 
median filter 17 depends on the quality of a CCD. That 
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is, for example, a median filter having a size of 5 pixels 
X 5 pixels is used for a CCD which causes much noise, and 
a median filter having a size of 3 pixels x 3 pixels is 
used for a CCD which causes small noise. 

Hereinafter, a principle of the present invention 
will be described in detail. 

As described above, when a high-range luminance 
component is emphasized by a compensation filter having 
a frequency characteristic represented by graph c shown 
in Figure 21, frequency components having a frequency 
higher than 1/2 of a sampling frequency f s are undesirably 
emphasized as well, whereby jaggy is caused. In order 
to remove such jaggy, according to the present invention, 
a middle-high range component represented by graph e of 
Figure 21 is mainly compensated for, and middle- and 
high-range components of an image are supplementarily 
compensated for. As shown by graph e of Figure 21, the 
middle-high range component has a maximum value of its 
amplitude at a position where the angular frequency co is 
between jt/2 to jr, and the amplitude thereof is zero when 
the angular frequency o is jt. 
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For the purpose of simplifying the description, 
a one -dimensional filter is used Instead of a two- 
dimensional filter In an example Illustrated below. 
A filter which can extract the middle-high luminance 
component is a filter (A) shown in Figure 6. The transfer 
function of the filter (A) of Figure 6 having a size of 
an even-number of pixels is as shown in expression (8). 
It should be noted that a center of the filter (A) of 
Figure 6 having a size of an even -number of pixels is 
present at an intermediate position between pixels, and 
a position of a filter output is also present at an 
intermediate position between pixels. The reason that 
a filter having a size of an even-number of pixels is used 
is to obtain a frequency characteristic represented by 
graph e of Figure 21. In the filter (A) of Figure 6, the 
number shown in each box represents a "weight" of a pixel 
corresponding to the box. 

H(z) = -Z-'-' + Z-°-' + Z°-^-Z^-' 
= 2cos(0.5a)) -2cos(1.5a)) ...(8) 
where z = e^^ = cosco+jsinco 

For the purpose of comparing the filter (A) with 
a middle-range luminance component compensation filter 
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and a high-range luminance component compensation filter 
which are used in a conventional technique, the two- 
dimensional second-derivative filter of Figure 17B is 
converted into a one-dimensional filter to prepare a 
filter (B) of Figure 6, and the two-dimensional 
second-derivative filter of Figure 17A is converted into 
a one-dimensional filter to prepare a filter (C) of 
Figure 6. The transfer function of the filter (B) of 
Figure 6 is as shown in expression (9), and the transfer 
function of the filter (C) of Figure 6 is as shown in 
expression ( 10 ) : 

H(z) = -Z-' + 2Z**-Z' 

= 2-2cos(co) ... (9) 

H(z) = -Z-^+2Z''-Z^ 

= 2-2cos(2co) . . . (10) 

The transfer functions represented toy 
expressions (8), (9), and (10) are normalized, and 
frequency characteristics thereof are shown in graphs A, 
B, and C, respectively, in Figure 7. The frequency 
characteristic of expression (8) shown in graph A 
exhibits a maximum value of the amplitude when the angular 
frequency co is 0.6jt, and exhibits an amplitude of zero 
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when the angular frequency co is jt. Thus, the compensation 
with middle-high luminance component can be achieved. 

In the case of using a filter having a size of an 
odd-number of pixels where a value (amplitude) of its 
transfer function is zero when the angular frequency co 
is jr, the amplitude thereof reaches a maximum value only 
when the angular frequency co is jt/2 (equivalent to fs/4) . 
For example, referring to Figure 8 which shows frequency 
characteristics of the filters (A), (D), and (E) of 
Figure 6, the amplitude of the frequency characteristic 
for each of the filters (D) and (E), i.e. , a filter having 
a size of an odd-number of pixels, is not zero when the 
angular frequency o) is jt. Thus, a filter having a size 
of an odd-number of pixels causes noise or jaggy in a 
reproduced image as in a conventional technique. 

Thus, for the purpose of compensating for a 
high-range component without emphasizing noise or jaggy, 
it is necessary to use a filter having a size of an 
even-number of pixels, such as the filter (A) of Figure 6. 
A filter (F) of Figure 6 has a size of an even-number of 
pixels which is obtained by adding a high-order derivative 
term to the filter (A). The transfer function of the 
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filter (F) is as shown in expression (11): 

H(z) = Z-2-5-5Z-'-' + 4Z-°'^ + 4Z°-'-5Z'-' + Z''-' 

= 8cos(0.5a))-10cos(1.5co) 

+ 2cos(2.5co) ... (11) 

Referring to Figure 9, the frequency 
characteristic of a normalized transfer function (8) 
(filter (A) of Figure 6) is represented by graph A, and 
the frequency characteristic of a normalized transfer 
function (11) (filter (F) of Figure 6) is represented by 
graph F. Graph A exhibits a peak of amplitude at a 
position where co = 0.6n:, and graph F exhibits a peak of 
amplitude at a position where co = O.SBTT. in both of 
graph A and graph F, the amplitude is zero at a position 
where 0) = TT . Thus , both of the filter (A) and filter (F) 
can be used to compensate for a middle-high range 
luminance component. However, the filter (F) is more 
useful than the filter (A) because, when graph A 
(filter (A)) and graph F (filter (F)) exhibit the maximum 
amplitude, the angular frequency co for graph F is larger 
than that for graph A. In other words, by using a filter 
which exhibits the amplitude of zero at a position where 
0) = TT , frequency components having a frequency higher 
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than fs/2 can be prevented from causing noise. 
Furthermore, within a angular frequency range of jt/2 to 
jr, as a value of angular frequency co at which the frequency 
characteristic exhibits a maximum value of amplitude 
becomes closer to iz, a higher-range luminance component 
can be compensated for. 

Furthermore, using a filter having a size of an 
even-number of pixels means that new pixels are created, 
not at original pixel positions of a CCD, but at positions 
on borders between original pixels . Thus , a two- 
dimensionally arranged filter has an arrangement shown 
in Figure 10. In Figure 10, symbols "©" denote original 
pixel positions of the CCD, and symbols "O" denote pixel 
data positions after a filter having a size of an 
even-number of pixels has been applied. 

For the purpose of examining an operation of an 
image processing apparatus of the present invention, the 
frequency characteristic as to luminance signal 
processing when a one-dimensional filter is used is now 
described. Referring to Figure 1, the description is 
made while referring to a series of processing shown in 
Figure 1 since the primary-color CCD area sensor 3 
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outputs image data until a luminance signal (YL+aYH) is 
obtained by the adder 8. Each process in the luminance 
generation section 5, the white balance adjustment 
section 13, the gamma correction section 14, and the 
low-frequency luminance signal generation section 15 
does not influence a frequency distribution of image data, 
and therefore, descriptions thereof are herein omitted. 

In the first RGB interpolation section 4, R- and 
B -components are interpolated by using a bilinear method 
of expressions (2) through (4), and a G-component is 
interpolated by using a median method of expression (5) . 
With such interpolation processing, a middle-high range 
component is attenuated, but is not completely removed. 

The middle-high range luminance component 
extraction section 6 uses the YH interpolation filters 
shown in Figures 2A and 2B to extract a middle -high range 
luminance component YH. The extracted middle-high range 
luminance component YH can be freely adjusted by using 
gain ct. It should be noted that, for the purpose of 
simplifying the description, a loss of the middle-high 
range luminance component YH in the first RGB 
interpolation section 4 is not considered. 
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The two-dimensional filters for extracting a 
middle-high range luminance component (first and second 
YH extraction filters Fl and F2 shown in Figures 2A 
and 2B) correspond to the one-dimensional filter (F) of 
Figure 6, the transfer function of the filter (F) is 
represented by expression (11), 

The second RGB interpolation section 12 uses, as 
a RGB interpolation filter, two-dimensional filters shown 
in Figures 5A and 5B. The two-dimensional filters shown 
in Figures 5A and 5B correspond to a one-dimensional 
filter (G) of Figure 6, the transfer function of the 
filter (G) is represented by expression (12): 

H( 303) = 2'^-^ + 3Z-°-^+3Z''^ + Z'-^ .,.(12) 

In Figure 11, the frequency characteristic of 
expression (11) of the YH extraction filter is 
represented by graph J, and the frequency characteristic 
of expression (12) of the RGB interpolation filter which 
is used in the second RGB interpolation section 12 is 
represented by graph I. (It should be noted that 
Figure 11 shows the normalized functions,) The 
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frequency characteristic of a luminance signal (YL+YH) 
is represented by graph H. From Figure 11, it is clearly 
understood that the middle-high range luminance 
component YH has been compensated for. 

Compensation of a middle-range luminance 
component and a high-range luminance component is 
supplementarily carried out by using the two filters shown 
in Figures 17A and 17B. For comparison. Figure 12 shows 
again the frequency characteristic of transfer 
function (11) of the YH extraction filters shown in 
Figures 2A and 2B which is in the form of a one- 
dimensional function, and the frequency characteristics 
of transfer functions (9) and (10) of the two two- 
dimensional second-derivative filters shown in 
Figures 17A and 17B which are in the form of a one- 
dimensional function. In Figure 12, graph J shows the 
frequency characteristic of filter (F) of Figure 6 which 
is represented by expression (11), graph B shows the 
frequency characteristic of the high-range luminance 
component extraction filter shown in Figure 17B, and 
graph C shows the frequency characteristic of the 
middle-range luminance component extraction filter shown 
in Figure 17A. Considering influence of the optical 
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low-pass filter 2 (Figure 20), a middle -range luminance 
component and a high-range luminance component in graph H 
show in Figure 11 can be compensated for by utilizing the 
frequency characteristics of graph B and graph C show in 
Figure 12. With only compensation with middle-high 
range component YH, a frequency characteristic cannot be 
compensated so as to be identical with the ideal 
characteristic represented by graph C of Figure 20. 
However, by adjusting parameters 1 and m of the composite 
contour emphasizing filter shown in Figure 4, gain a and 
ENH gain p, a frequency characteristic which is closer 
to the ideal characteristic represented by graph C can 
be obtained. 

As described hereinabove, according to this 
embodiment of the present invention, as shown in Figure 1, 
an image processing system includes a middle-high range 
luminance component YH generation route including the 
first RGB interpolation section 4 and the middle-high 
range luminance component extraction section 6 in 
parallel with a low-frequency luminance signal YL 
generation route including the second RGB interpolation 
section 12, and the middle-high range luminance 
component YH is added by the adder 8 to low- frequency 
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luminance signal YL, whereby a middle -high range 
luminance component is compensated for by the middle- 
high range luminance component YH. Furthermore, a 
middle-range luminance component and a high-range 
luminance component are supplementarily compensated for 
by using a conventional compensation method. Thus, an 
image with higher resolution can be obtained as compared 
with an image obtained by a conventional image processing 
system, while noise and jaggy can be prevented from 
appearing in a reproduced image. 

According to this embodiment of the present 
invention, in order to prevent occurrence of false color, 
a color-difference signal is generated from a second RGB 
image signal which is generated by interpolation with a 
low-pass filter in the second RGB interpolation 
section 12 {Figure 1) , and then is subjected to the median 
filter 17 (Figure 1) before it is externally output . On 
the other hand, in order to obtain a desirable luminance 
signal, the middle-high range luminance component YH is 
added by the adder 8 to low- frequency luminance signal YL, 
and a middle-range luminance component and high-range 
luminance component in a resultant signal are 
supplementarily compensated. When the middle-high range 



- 54 - 



01R00269 



luminance component YH is added to low-frequency 
luminance signal YL in ttie adder 8, the position of image 
data has to be adjusted. As described above, in the 
middle-high range luminance component extraction 
section 6, for the purpose of extracting middle-high 
range luminance component YH, an extraction filter used 
must be a filter having a size of an even-number of pixels. 
Accordingly, the first RGB interpolation section 4 has 
to use, as a RGB interpolation filter, a filter having 
a size of an odd-number of pixels, especially a filter 
having a size of 3 pixels x 3 pixels, which is the best 
size in preventing deterioration of middle-high range 
luminance component YH. Furthermore, the second RGB 
interpolation section 12 has to use, as a RGB 
interpolation filter, a filter having a size of an 
even-number of pixels . 

In the present embodiment, the color filter 
provided to the CCD area sensor is a color filter whose 
RGB color arrangement based on the Bayer array. However, 
the RGB color arrangement which can be used in the present 
invention is not limited to the Bayer array, but any RGB 
color arrangement may be used. 
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The above descriptions as to an image processing 
apparatus of the present invention is now summarized. 
Referring to Figure 1, the image processing apparatus 1 
processes RGB image data generated by the optical low-pass 
5 filter 2 and the area sensor 3. In this image processing, 
a low- frequency luminance signal generated from the RGB 
image data has a middle-high range luminance component 
which is attenuated due to the optical low-pass filter 2 
as compared with an ideal frequency characteristic which 

10 should be obtained by the optical low-pass filter 2, The 
image processing apparatus 1 includes means of 
compensating for the attenuated middle-high range 
luminance component of the low-frequency luminance signal. 
Owing to such compensation means, a sharper image can be 

15 obtained by contour emphasizing processing while 
preventing occurrence of noise and jaggy which may be 
caused during the contour emphasizing processing. 

According to the present invention, a middle-high 
20 range luminance component of a low- frequency luminance 
signal which is attenuated as compared with an ideal 
frequency characteristic in a range of a predetermined 
frequency (a 1/2 of sampling frequency fs) or smaller is 
compensated. Thus, a sharp image can be obtained by 
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contour emphasizing processing while preventing 
occurrence of noise and jaggy which may be caused when 
obtaining the sharp image. 

According to the present invention, an image with 
high resolution can be obtained by contour emphasizing 
processing while preventing occurrence of noise and jaggy 
which may be caused when obtaining the sharp image. 

According to the present invention, a middle-high 
range component whose amplitude is zero when the angular 
frequency co is jt and has a maximum value at a position 
where the angular frequency co is between jt/2 to jt can be 
readily extracted. 

According to the present invention, a uniform 
filtering effect can be obtained, and as a result, an image 
can be faithfully reproduced. 

According to the present invention, arithmetic 
operations in x- and y-directions of image data can be 
separately performed, and accordingly, increases in the 
amount of arithmetic operations can be suppressed. Thus, 
YH extraction filtering can be readily implemented by 
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hardware . 

According to the present Invention, a middle-high 
range luminance component can be extracted while 
most -effectively preventing deterioration of the 
middle-high range luminance component. 

According to the present invention, when 
compensating for a middle-high range luminance component 
of a low-frequency luminance signal, a center of the 
middle-high range luminance component is present at a 
position between neighboring pixels, and a center of the 
low- frequency luminance signal is also present at a 
position between neighboring pixels. Thus, occurrence 
of a ghost in a reproduced image can be prevented. 

According to the present invention, a G-component 
is interpolated by using a median method, whereby 
attenuation of a high-range luminance component is 
suppressed to a minimum level. R- and B-components are 
interpolated by using a bilinear method, whereby noise 
is reduced. Thus, a contour of an image is emphasized, 
and the quality of the image can be improved. 
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According to the present invention, by changing 
a ratio between a middle -range luminance component and 
a high-range luminance component, the three-dimensional 
appearance (stereoscopic effect or stereophonic effect) 
5 of an image can be adjusted according to user's 
preference. 

According to the present invention, a 
middle/high-range component extraction section can be 
10 readily formed of a single filter. 

According to the present invention, the amount of 
noise included in a color-difference signal varies 
according to, for example, the quality of an image 
15 capturing element such as a CCD. Thus, by selecting an 
appropriate median filter according to the amount of noise, 
a color-difference signal with reduced noise can be 
generated. 

20 Various other modifications will be apparent to 

and can be readily made by those skilled in the art without 
departing from the scope and spirit of this invention. 
Accordingly, it is not intended that the scope of the 
claims appended hereto be limited to the description as 
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set forth herein, but rather that the claims be broadly 
construed. 



